Research Methods in Developmental Linguistics – Week 6

Dr Stefano Coretta

University of Edinburgh

Song 2020

Note

• Data from Song 2020.

• Shallow morphological representation in L2 speakers.

• Lexical decision task: Is the word a real English word or not?

• Priming paradigm:

  • Prime: prolong (unrelated), unkind (constituent), kindness (non-constituent).

  • Target: unkindness ([un-kind]-ness, not un-[kind-ness]).

• English L1 and English L2/Cantonese L1 participants.

Song 2020: hypotheses

Warning

Vague hypothesis:

The Shallow Structure Hypothesis (SSH) allows the possibility for nonnative speakers to show strong priming effects from nonconstituent and constituent primes equally because the shallow processing route, which relies more heavily on semantic and surface-level information than on structural information.

Song 2020: the data

shallow <- read_csv("data/song2020/shallow.csv") |>
  filter(
    Critical_Filler == "Critical",
    Branching == "Left"
  ) |> 
  mutate(
    accuracy = ifelse(ACC == 0, "incorrect", "correct"),
    accuracy = factor(accuracy, levels = c("incorrect", "correct"))
  )

Rows: 6500 Columns: 11
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (8): Group, ID, List, Target, Critical_Filler, Word_Nonword, Relation_ty...
dbl (3): ACC, RT, logRT

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

shallow

Song 2020: accuracy

Bernoulli regression (accuracy)

my_seed <- 4235

sha_bm_1 <- brm(
  accuracy ~ 0 + Group:Relation_type,
  family = bernoulli,
  data = shallow,
  cores = 4,
  seed = my_seed,
  file = "data/cache/sha_bm_1"
)

Compiling Stan program...

Start sampling

sha_bm_1

Bernoulli regression: expected predictions

conditional_effects(sha_bm_1, "Group:Relation_type")

BUT…

• Multiple observations from different participants.

• Multiple observations from different target words.

Tip

• We need to include varying terms.

By-subject accuracy

By-target accuracy

By-subject and by-target varying terms

sha_bm_2 <- brm(
  ACC ~ 0 + Group:Relation_type +
    (0 + Relation_type | ID) +
    (0 + Group:Relation_type | Target),
  family = bernoulli,
  data = shallow,
  cores = 4,
  seed = my_seed,
  file = "data/cache/sha_bm_2"
)

Compiling Stan program...

Start sampling

summary(sha_bm_2)

tion_typeNonConstituent)        0.34
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)        0.31
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        0.28
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        0.58
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeNonConstituent)     0.37
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             0.42
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             0.56
cor(GroupL1:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)          0.44
cor(GroupL2:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)          0.43
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             0.30
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             0.64
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)          0.43
cor(GroupL2:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)          0.49
cor(GroupL1:Relation_typeUnrelated,GroupL2:Relation_typeUnrelated)               0.57
                                                                             Est.Error
sd(GroupL1:Relation_typeConstituent)                                              0.64
sd(GroupL2:Relation_typeConstituent)                                              0.40
sd(GroupL1:Relation_typeNonConstituent)                                           0.45
sd(GroupL2:Relation_typeNonConstituent)                                           0.30
sd(GroupL1:Relation_typeUnrelated)                                                0.36
sd(GroupL2:Relation_typeUnrelated)                                                0.30
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeConstituent)            0.27
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)         0.27
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)         0.25
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)         0.29
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)         0.22
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeNonConstituent)      0.27
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)              0.25
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)              0.21
cor(GroupL1:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)           0.25
cor(GroupL2:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)           0.27
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)              0.25
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)              0.20
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)           0.24
cor(GroupL2:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)           0.24
cor(GroupL1:Relation_typeUnrelated,GroupL2:Relation_typeUnrelated)                0.22
                                                                             l-95% CI
sd(GroupL1:Relation_typeConstituent)                                             0.53
sd(GroupL2:Relation_typeConstituent)                                             0.79
sd(GroupL1:Relation_typeNonConstituent)                                          0.67
sd(GroupL2:Relation_typeNonConstituent)                                          0.37
sd(GroupL1:Relation_typeUnrelated)                                               0.66
sd(GroupL2:Relation_typeUnrelated)                                               0.74
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeConstituent)          -0.31
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)       -0.23
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)       -0.21
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)       -0.31
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        0.04
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeNonConstituent)    -0.19
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)            -0.13
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             0.07
cor(GroupL1:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)         -0.10
cor(GroupL2:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)         -0.16
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)            -0.22
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             0.18
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)         -0.08
cor(GroupL2:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)         -0.04
cor(GroupL1:Relation_typeUnrelated,GroupL2:Relation_typeUnrelated)               0.08
                                                                             u-95% CI
sd(GroupL1:Relation_typeConstituent)                                             3.01
sd(GroupL2:Relation_typeConstituent)                                             2.36
sd(GroupL1:Relation_typeNonConstituent)                                          2.39
sd(GroupL2:Relation_typeNonConstituent)                                          1.55
sd(GroupL1:Relation_typeUnrelated)                                               2.07
sd(GroupL2:Relation_typeUnrelated)                                               1.89
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeConstituent)           0.72
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)        0.81
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)        0.76
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        0.78
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        0.91
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeNonConstituent)     0.82
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             0.84
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             0.89
cor(GroupL1:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)          0.84
cor(GroupL2:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)          0.86
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             0.75
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             0.92
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)          0.82
cor(GroupL2:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)          0.88
cor(GroupL1:Relation_typeUnrelated,GroupL2:Relation_typeUnrelated)               0.90
                                                                             Rhat
sd(GroupL1:Relation_typeConstituent)                                         1.00
sd(GroupL2:Relation_typeConstituent)                                         1.00
sd(GroupL1:Relation_typeNonConstituent)                                      1.00
sd(GroupL2:Relation_typeNonConstituent)                                      1.00
sd(GroupL1:Relation_typeUnrelated)                                           1.00
sd(GroupL2:Relation_typeUnrelated)                                           1.00
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeConstituent)       1.00
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)    1.00
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)    1.00
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)    1.00
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)    1.00
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeNonConstituent) 1.00
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)         1.00
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)         1.00
cor(GroupL1:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)      1.00
cor(GroupL2:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)      1.00
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)         1.00
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)         1.00
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)      1.00
cor(GroupL2:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)      1.00
cor(GroupL1:Relation_typeUnrelated,GroupL2:Relation_typeUnrelated)           1.00
                                                                             Bulk_ESS
sd(GroupL1:Relation_typeConstituent)                                             1795
sd(GroupL2:Relation_typeConstituent)                                             2425
sd(GroupL1:Relation_typeNonConstituent)                                          2709
sd(GroupL2:Relation_typeNonConstituent)                                          3228
sd(GroupL1:Relation_typeUnrelated)                                               3705
sd(GroupL2:Relation_typeUnrelated)                                               3833
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeConstituent)           1757
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)        2678
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)        3001
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        2560
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        3456
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeNonConstituent)     4272
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             2504
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             3901
cor(GroupL1:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)          3867
cor(GroupL2:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)          3674
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             2401
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             3356
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)          3950
cor(GroupL2:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)          3507
cor(GroupL1:Relation_typeUnrelated,GroupL2:Relation_typeUnrelated)               3302
                                                                             Tail_ESS
sd(GroupL1:Relation_typeConstituent)                                             2343
sd(GroupL2:Relation_typeConstituent)                                             3228
sd(GroupL1:Relation_typeNonConstituent)                                          3029
sd(GroupL2:Relation_typeNonConstituent)                                          2684
sd(GroupL1:Relation_typeUnrelated)                                               3014
sd(GroupL2:Relation_typeUnrelated)                                               3401
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeConstituent)           2539
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)        2900
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeNonConstituent)        3094
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        3065
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeNonConstituent)        3434
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeNonConstituent)     3507
cor(GroupL1:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             3101
cor(GroupL2:Relation_typeConstituent,GroupL1:Relation_typeUnrelated)             3625
cor(GroupL1:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)          3383
cor(GroupL2:Relation_typeNonConstituent,GroupL1:Relation_typeUnrelated)          3577
cor(GroupL1:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             3297
cor(GroupL2:Relation_typeConstituent,GroupL2:Relation_typeUnrelated)             3393
cor(GroupL1:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)          3652
cor(GroupL2:Relation_typeNonConstituent,GroupL2:Relation_typeUnrelated)          3410
cor(GroupL1:Relation_typeUnrelated,GroupL2:Relation_typeUnrelated)               3255

Regression Coefficients:
                                    Estimate Est.Error l-95% CI u-95% CI Rhat
GroupL1:Relation_typeConstituent        2.77      0.62     1.69     4.12 1.00
GroupL2:Relation_typeConstituent        2.01      0.50     1.12     3.08 1.00
GroupL1:Relation_typeNonConstituent     1.80      0.49     0.91     2.87 1.00
GroupL2:Relation_typeNonConstituent     1.52      0.36     0.86     2.30 1.00
GroupL1:Relation_typeUnrelated          1.45      0.42     0.68     2.34 1.00
GroupL2:Relation_typeUnrelated          0.99      0.38     0.28     1.77 1.00
                                    Bulk_ESS Tail_ESS
GroupL1:Relation_typeConstituent        2552     2703
GroupL2:Relation_typeConstituent        2519     2407
GroupL1:Relation_typeNonConstituent     2789     2924
GroupL2:Relation_typeNonConstituent     3126     2871
GroupL1:Relation_typeUnrelated          3313     2920
GroupL2:Relation_typeUnrelated          2723     2969

Draws were sampled using sampling(NUTS). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).

By-subject and by-target model: expected predictions plot

Song 2020: hypotheses

Warning

Vague hypothesis:

The Shallow Structure Hypothesis (SSH) allows the possibility for nonnative speakers to show strong priming effects from nonconstituent and constituent primes equally because the shallow processing route, which relies more heavily on semantic and surface-level information than on structural information.

Difference between non/constituent in L2

sha_bm_2_draws <- as_draws_df(sha_bm_2, variable = "b_", regex = TRUE)

sha_bm_2_draws <- sha_bm_2_draws |> 
  mutate(
    diff_l2 = `b_GroupL2:Relation_typeNonConstituent` - `b_GroupL2:Relation_typeConstituent`
  )

sha_bm_2_draws

Difference between non/constituent in L2: plot

Difference between non/constituent in L2: CrIs

sha_bm_2_draws |> 
  select(diff_l2) |> 
  summarise_draws(~quantile(.x, probs = c(0.05, 0.95, 0.15, 0.85))) |> 
  mutate(
    across(where(is.numeric), ~round(.x, digits = 2))
  )

Warning: Dropping 'draws_df' class as required metadata was removed.

Expected predictions by subject: get draws of constant terms

sha_bm_2_draws <- as_draws_df(sha_bm_2, variable = c("^b_.*"), regex = TRUE) |> 
  pivot_longer(
    `b_GroupL1:Relation_typeConstituent`:`b_GroupL2:Relation_typeUnrelated`,
    names_to = c("Group", "Relation_type"),
    names_pattern = "b_Group(.*):(.*)"
  )

Warning: Dropping 'draws_df' class as required metadata was removed.

sha_bm_2_draws

Expected predictions by subject: get draws of varying terms

sha_bm_2_r <- spread_draws(sha_bm_2, r_ID[ID,Relation_type]) |> 
  mutate(
    Group = str_sub(ID, 1, 2)
  )

sha_bm_2_r

Expected predictions by subject: join

sha_bm_2_draws_r <- left_join(sha_bm_2_draws, sha_bm_2_r) |> 
  mutate(
    epred = value + r_ID
  )

Joining with `by = join_by(.chain, .iteration, .draw, Group, Relation_type)`

sha_bm_2_draws_r

Expected predictions by subject: plot

Difference between expected predictions by-subject

sha_bm_2_draws_r_wide <- sha_bm_2_draws_r |>
  select(-value, -r_ID) |> 
  pivot_wider(names_from = Relation_type, values_from = epred) |> 
  mutate(
    ncon_con = Relation_typeNonConstituent - Relation_typeConstituent
  )

sha_bm_2_draws_r_wide

Difference between expected predictions by-subject: plot

Results overview

Note

The Shallow Structure Hypothesis (SSH) allows the possibility for nonnative speakers to show strong priming effects from nonconstituent and constituent primes equally because the shallow processing route, which relies more heavily on semantic and surface-level information than on structural information.

• The 90% CrI of the difference between nonconstituent and constituent in L2 is [-1.3, 0.26].

• At 70% probability, the log-odds of the priming effect on accuracy between nonconstituent and constituent in L2 participants differs between -0.98 and -0.01.

• There is a lot of uncertainty in the expected predictions of individual participants, although there is a tendency for negative differences between nonconstituent and constituent.

Summary

Tip

• You should include varying terms if data is “hierarchical”, for example repeated measures from participants or items.

• Not including varying terms (wrongly) inflates posterior certainty.

• Frequentist regression models with lme4 often don’t converge and researchers simplify the hierarchical structure of the model.